Repository navigation
benchmark(opus-5-5): Opus 5.5 on Bedrock via strands and omp, v2 dataset - #195
Merged
Merged
Conversation
added 2 commits
October 2, 2026 16:56
21 tasks, all scored, mean 81.21, mean cost $2.46/task. First benchmark of the Strands Agents harness (aarora79#194) on a full dataset.
21 tasks, all scored, mean 81.16, mean cost $4.44/task. Direct harness comparison with the strands run (81.21 at $2.46/task): same quality, strands costs 45% less from prompt caching.
aarora79
added a commit
that referenced
this pull request
Oct 5, 2026
…nt Path 2 runs as metered (#210) * Add Opus 5.5 and Kimi K3 on Bedrock to the omp/swe3 frontier Regenerate the omp/swe3 v2 frontier JSON, the cost-quality, bubble and radar charts (light and dark) and the omp report table over every omp v2 run on disk. Two runs had landed since the last regeneration: claude-opus-5-5 from #195 (81.16 at $4.44 a task) and kimi-k3-bedrock from #191 (72.87 at $3.87). Opus 5.5 joins both the combined and the Bedrock frontiers. It scores higher than claude-sonnet-5 (76.97 at $4.67) and glm-5.3 (80.11 at $7.04) for less, so those two leave the frontier. kimi-k3-bedrock is dominated by qwen3.8-27b. Add both models to swe-router's model-aliases.json. * Rebuild swe-router's models.json and update the frontier claims Rebuild vend/swe-router/models.json from the new omp/swe3 frontier (21 models). Rewrite the README and results-index sentences the new frontier made wrong: claude-opus-5-5 (81.16 at $4.44) now sits behind claude-opus-5 in place of glm-5.3, and replaces claude-sonnet-5 on the Bedrock frontier. Update the model counts that describe the chart and models.json. * Ship claude-opus-5-5 in place of glm-5.3 in swe-router's allow-list The shipped allow-list is the combined cost/quality frontier, which test_route enforces. Opus 5.5 (81.16, $4.44) replaced glm-5.3 (80.11, $7.04) on that frontier. Update the comments that counted self-hosted entries and explained why claude-sonnet-5 is absent. * Count a Bedrock model reached through LiteLLM as metered on the charts plot_cost_quality.py and gen_agent_report.py decided a run was metered only when its provider was "bedrock". A Path 2 run, a Bedrock model reached through the LiteLLM proxy, records "endpoint", so kimi-k3-bedrock was drawn and grouped as self-hosted and its report row showed no cost. Treat an endpoint run as metered when no throughput sweep prices it and it reports a positive bill. An audit of all 57 endpoint run summaries finds two such runs, kimi-k3-bedrock (omp) and qwen.qwen3-coder-30b-a3b-instruct (codex), both Bedrock-billed; every self-hosted run is unchanged. kiro-cli credits stay their own basis. Regenerate the omp/swe3 charts, frontier JSON and report. kimi-k3-bedrock (72.87 at $3.87) joins the Bedrock frontier and claude-opus-4-5 (66.32 at $4.18) leaves it. The combined frontier is unchanged. * Rebuild models.json and describe the new Bedrock frontier Rebuild swe-router's models.json after kimi-k3-bedrock moved to the metered basis. In the results index, list kimi-k3-bedrock on the Bedrock frontier in place of claude-opus-4-5, and say Path 2 now has a published run.
aarora79
added a commit
that referenced
this pull request
Oct 10, 2026
omp's --model is a fuzzy match. A Bedrock id missing from the catalog omp has loaded runs on the nearest model it knows, and omp reports no error. omp 18.1.15 had no Claude 5.5 entries, so a 21-task "Haiku 5.5" run on 2026-10-08 was Haiku 4.5 on every turn. Upgrading omp is not enough: in a fresh agent dir, omp -p resolves against the snapshot bundled in the binary, and 18.8.8 still sends Haiku 5.5 to Haiku 4.5 until `omp models` has fetched and cached the full catalog in that dir. run-swe-headless.py now: - exits before the first task when `omp models amazon-bedrock --json` does not list the exact Bedrock id, and says to run `omp update`; - runs the same listing in each Bedrock task's own agent dir, which warms the catalog that omp -p then reads; - compares the model on every assistant message with the one requested, fails a mismatch as the non-retryable `error_model_mismatch`, and records the ids in metrics.json as `models_used`. run-swe-router-headless.py applies the per-dir warm-up and the stream check to each omp judgment, and raises on a mismatch. A second bug surfaced in the same run. omp can end a finished session on stopReason "toolUse"; the harness counted that as an error, retried, and the retry deleted all six finished artifacts before producing nothing. A task with every artifact on disk is no longer retried. The published Opus 5.5 omp row (#195) is unaffected: a hello-world re-run on omp 18.8.8 answered as us.anthropic.claude-opus-5-5 on every turn, and its omp-reported cost matches Opus 5.5 rates to the cent, the same rates #195's $93.27 matches. A fallback to Opus 5 would have cost $157 to $173.
aarora79
added a commit
that referenced
this pull request
Oct 11, 2026
claude-opus-5-5/omp/swe3/Hello-World: the one-task check that settled whether the published Opus 5.5 omp row ran Opus 5.5. On omp 18.8.8 every turn named us.anthropic.claude-opus-5-5 (models_used), it scored 91.6, and its omp-reported $1.1855 matches Opus 5.5 rates to the cent, the same rates #195's $93.27 matches. Hello-World runs commit their full artifacts by design; the raw event stream stays ignored. Two runs go under do-not-include/, which every generator's one-level scan skips, with a README section each: - claude-haiku-5-5-resolved-as-haiku-4-5: the 2026-10-08 run requested as Haiku 5.5 that omp 18.1.15 fuzzy-matched to Haiku 4.5 on every turn (21 generated, 13 judged, 55.58). The evidence behind #218. - claude-haiku-5-5-misplaced-artifacts: the six artifacts Haiku 5.5 wrote for registration-admission-control-gate under the v1 folder instead of the v2 path it was given. Never judged; the task stays a failure in the published run. Added with -f, since .gitignore keeps run markdown out by default. One hunk header in its patch quoted an example token from upstream's terraform.tfvars.example; it is redacted. The pre-commit hook also stripped trailing whitespace from 25 of its lines; the patch still applies at v1.0.19. The charts and models.json still see 23 models; Haiku 5.5 stays at 72.69.
aarora79
added a commit
that referenced
this pull request
Oct 11, 2026
… v2 dataset (#219) * benchmark(haiku-5-5, sonnet-5-5): Claude 5.5 on Bedrock via omp/swe3, v2 dataset Both ran on omp 18.8.8 with the model checks from #218; every assistant message in every task names the requested model (models_used). claude-sonnet-5-5 79.49 $2.62/task 21/21 scored claude-haiku-5-5 72.69 $0.14/task 20/21 scored Haiku 5.5 failed registration-admission-control-gate: it wrote all six artifacts under mcp-gateway-registry/ instead of the mcp-gateway-registry-v2/ path the prompt named 56 times, and its retry wrote nothing. It counts as a failed task, as for any other model. Both join the combined and the Bedrock frontiers. On the combined frontier they displace kimi-k3, gemma-4-31b and qwen3.6-35b; on the Bedrock frontier, claude-haiku-4-5 and kimi-k3-bedrock. Add Claude Haiku 5.5 to bedrock_pricing.py from the AWS price list (AmazonBedrockFoundationModels, us-east-1): $0.11 input, $0.55 output, $0.011 cache read, $0.1375 cache write per million tokens, regional. omp prices runs from its own catalog, which carries the same rates. Regenerate the omp/swe3 frontier JSON, the cost-quality, radar and bubble charts (light and dark) and the omp report. Update the README and results-index frontier claims, and correct the README badge from "45 models" (a June count of models reachable on Bedrock) to the 28 models with committed benchmark results, from 11 providers. * Rebuild swe-router for the Claude 5.5 frontier Rebuild vend/swe-router/models.json from the new omp/swe3 frontier: 23 models, measured 2026-10-11, source commit 7e8657f. The shipped allow-list is the combined frontier, which test_route enforces. It becomes claude-opus-5, claude-opus-5-5, claude-sonnet-5-5, qwen3.8-27b and claude-haiku-5-5. kimi-k3, gemma-4-31b and qwen3.6-35b leave it and move to the commented suggestions for teams that self-host. Only one of the five is now self-hosted, so the file's warning says so. Add aliases for claude-sonnet-5-5 and claude-haiku-5-5. Regenerate the README's four per-tier tables from models.json; they had listed 18 models and now list all 23, and every existing row is unchanged. Add both models to SKILL.md's tier comparison: Haiku 5.5 scores 79.2 on low and 60.0 on high, with one hard task unfinished. Update the route tests that encode the frontier: low work at a floor of 75 now goes to claude-haiku-5-5, high work at 75 still rejects both it and qwen3.8-27b. * Keep the Opus 5.5 hello-world check and the two excluded Haiku runs claude-opus-5-5/omp/swe3/Hello-World: the one-task check that settled whether the published Opus 5.5 omp row ran Opus 5.5. On omp 18.8.8 every turn named us.anthropic.claude-opus-5-5 (models_used), it scored 91.6, and its omp-reported $1.1855 matches Opus 5.5 rates to the cent, the same rates #195's $93.27 matches. Hello-World runs commit their full artifacts by design; the raw event stream stays ignored. Two runs go under do-not-include/, which every generator's one-level scan skips, with a README section each: - claude-haiku-5-5-resolved-as-haiku-4-5: the 2026-10-08 run requested as Haiku 5.5 that omp 18.1.15 fuzzy-matched to Haiku 4.5 on every turn (21 generated, 13 judged, 55.58). The evidence behind #218. - claude-haiku-5-5-misplaced-artifacts: the six artifacts Haiku 5.5 wrote for registration-admission-control-gate under the v1 folder instead of the v2 path it was given. Never judged; the task stays a failure in the published run. Added with -f, since .gitignore keeps run markdown out by default. One hunk header in its patch quoted an example token from upstream's terraform.tfvars.example; it is redacted. The pre-commit hook also stripped trailing whitespace from 25 of its lines; the patch still applies at v1.0.19. The charts and models.json still see 23 models; Haiku 5.5 stays at 72.69. * Fix a flaky stray-file sweep test on coarse filesystem clocks test_untracked_file_written_during_the_level_is_quarantined failed intermittently on GitHub runners (run 38108751467): it records since = time.time() and then writes a file, but the kernel stamps mtime from a coarser clock, so the file can carry an mtime a few milliseconds before since. _sweep_stray_root_writes skipped any file with st_mtime < since, so the stray was left in place. The same rounding would let a real sweep miss a stray written in the first milliseconds of a level, so fix the check, not the test: accept mtimes up to _MTIME_SLACK_SECONDS (2.0) before the level start. This cannot widen what is moved, because a candidate must also be absent from the snapshot taken at the level start, a plain file, and untracked. Add a test that backdates a new file 50 ms before since; it fails with no slack and passes with it.
3 tasks done
aarora79
added a commit
that referenced
this pull request
Oct 17, 2026
… dataset (#229) Claude Opus 5.5 (us.anthropic.claude-opus-5-5) under --agent strands-harness, the Strands team's create_harness() agent, on the 21 mcp-gateway-registry-v2 tasks. Judged by openai.gpt-5.6-sol at high effort. All 21 tasks produced six artifacts on the first attempt, and the loop guard never fired. strands-harness 80.80 $2.42/task 21/21 scored strands 81.21 $2.46/task 21/21 scored (#195) omp 81.16 $4.44/task 21/21 scored (#195) The three harnesses land within 0.41 points of each other, and both Strands builds cost about 55% of omp per task. Regenerate harness-strands-harness-swe3.md with gen_agent_report.py and its frontier JSON and three charts, and restore the hand-written notes with an Opus 5.5 row. Add the row to docs/strands-harness.md, and fill in Sonnet 5.5's omp result (79.49, $2.62, from #219), which both pages still listed as not run. The combined chart covers Claude Code and pi on v1, and models.json comes from the omp frontier, so neither changes.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
--agent strands): mean 81.21, mean cost $2.46/task--agent omp): mean 81.16, mean cost $4.44/taskHeadline numbers
What is included
86 files across two commits:
benchmarks/swe-benchmark-data/claude-opus-5-5/strands/swe3/mcp-gateway-registry-v2/(43 files)benchmarks/swe-benchmark-data/claude-opus-5-5/omp/swe3/mcp-gateway-registry-v2/(43 files)Each commit adds
eval.json+metrics.jsonfor all 21 tasks, plus the aggregatedrun-summary.json.Test plan
run-summary.jsonwritten bysummarize_run.pyfor both runs